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APPENDIX  D 


A  UNIFYING  SET  OF  HYPOTHESES  FOR  DYNAMIC  SYSTEM  TEST  AND 
EVALUATION:  THE  RATING  AND  MEASUREMENT  OF  SYSTEM  PERFORMANCE, 
SYSTEM  LOAD,  AND  SYSTEM  WORK  AND  THEIR  INTERRELATIONSHIPS. 


SUMMARY 


A  set  of  equations,  definitions,  symbols,  units  of  measure¬ 
ment  and  an  equal  interval  scale  providing  an  equal  interval  unit 
rating  and  measurement  method  for  system  evaluation  are  presented 


•  V 


v  The  objective  is  to  provide  a  unified  approach  to  the  systems 
engineering  evaluation,  rating  and  measurement  of  a  dynamic  system. 


The  equations  present  the  hypothesized  relationships  between 
system  ratingsand  system  measures  of  performance,  work,  and  load. 


An  equal  interval,  one-tenth  power  of  ten,  scale  with  ten  as 
the  highest  rating  is  introduced  for  the  rating,  measuring,  and 
communicating  system  performance,  work,  and  load  factors.  These 
factors  are  of  increasing  interest  during  early  RfcD  system  dynamic 
simulation  concept  testing  and  later  operational  system  test  and 
evaluation.  4- — - - - 


INTRODUCTION 

The  following  definitions,  symbols,  and  equations  are  proposed 
to  provide  a  common  method  of  communicating  the  complex  relationship 


between  ratings  and  measures  of  system  performance,  system  load, 
and  system  work  during  dynamic  system  test  and  evaluation. 


DEFINITIONS 

System:  A  system  exists  in  its  simplest  form  when  one  person 
uses  a  piece  of  equipment  to  perform  a  useful  function. 

System  Evaluation:  The  functioning  of  the  system  is  assessed 
against  system  criteria  to  determine  qualitatively  and 
quantitatively  the  degree  to  which  the  operating  system 
meets  these  standards. 


System  Evaluation  Methodology:  The  design,  conduct,  and  analysis 
of  dynamic  human  operated  equipment  tests  so  as  to  provide  a 
true  assessment  of  the  systems  functional  utility . 


Systems  Performance:  A  summation  of  the  ratings  and  measures  of 
system  output  which  correlate  highly  with  systems  criteria. 
Systems  performance  is  hypothesized  to  be  qualitatively  and 
quantitatively  equal  to  and  varies  inversely  with  system  load 
and  system  work. 


System  Load:  The  input  load  imposed  on  the  system  during  operation 
designed  to  accomplish  its  useful  function.  It  is  hypothesized 
to  be  qualitatively  and  quantitatively  equal  to  and  varies 
directly  with  system  work;  therefore,  the  relationship  explains 


the  probable  origin  of  the  misnomer  in  the  venacular  of 
everyday  slang  and  common  usage  of  the  term  "workload". 

System  Work:  Not  only  the  useful  output  of  system  equipment  as 
defined  in  physics,  but  the  effort  required  of  a  person  to 
handle  the  system  load.  It  may  correlate  with  physiological 
measures  such  as  heartrate.  It  is  hypothesized  to  be 
qualitatively  and  quantitatively  equal  to  and  varies  directly 
with  system  load;  therefore,  this  relationship  explains  the 
probable  origin  of  the  misnomer  in  the  venacuJLar  oT' everyday 
slang  and  common  usage  of  the  term  "workload" . 


System  Criteria:  The  statement  of  the  ultimate  standards  by  which 
a  systems  performance  is  judged  and  measured.  Generally,  a 
statement  of  the  systems  goals.  For  example,  the  safe,  orderly 
and  efficient  control  of  air  traffic  used  as  the  criteria  for 
evaluating  an  Air  Traffic  Control  System. 

Baseline  or  Standard  Conditions:  The  optimum  rating  and  measure 
of  system  performance  obtained  under  conditions  of  optimum 
system  load  and  optimum  system  work.  It  is  quantified  as 
being  100  percent,  zero  decivals,  and  a  system  evaluation 
rating  of  ten,  10. 

Decival,  dv:  A  decival  is  a  proposed  system  evaluation  unit  of 
measurement  which  is  10  times  the  logarithm  of  equal  interval 
system  differences  which  are  plus  or  minus  one-tenth  powers 
of  ten  apart. 


Decival,  dv  (Continued) 

1  dv  =  10  log  [lO°«D  and  -1  dv  =  10  log  £o”°*3. 

This  change  is  equal  to  one  rating  unit  along  the  one-tenth 
power  of  ten  scale  shown  in  figure  1. 

Centival,  cv:  A  centival  is  a  system  evaluation  unit  of  measure¬ 
ment  which  is  10  times  the  logarithm  of  the  equal  interval 
distances  which  are  one-hundredth  powers  of  ten. 

1  cv  *  10  log  Jj0°‘®3  and  "1  cv  =  10  log  Qo”0,03l. 

A  System  Factor  Rateable  Difference:  The  hypothesized  smallest 
noticeable  difference  in  system  performance/  work,  and  load 
which  approximates  a  measured  change  of  one  centival.  This 
change  is  equal  to  one-tenth  of  a  rating  unit  along  the 
one-tenth  power  of  ten  scale  shown  in  figure  1 . 

System  performance,  work,  and  load  factors:  Those  system  measures 
which  are  proven  by  operational  test  results  to  correlate 
highly  with  magnitude  estimation  ratings  of  system  performance, 
work,  and  load.  These  ratings  may  be  obtained  from  operators 
of  the  systems  and/or  expert  operators  observing  the  system 
operation  and  using  an  equal  interval  (10° #1)  rating  scale 
shown  in  figure  1 . 

System  performance,  work,  and  load  measures:  Twenty-eight 

operationally  defined  and  measurable  changes  in  air  traffic 
control  system  performance  are  discussed  in  reference  no. 2, 
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The  FAA  Handbook  for  Experimenters.  They  are  formed  to 
fall  into  clusters  of  measures  called  factors  such  as 
the  number  and  duration  of  conflicts,  delays,  communications, 
etc.,  which  correlate  highly  with  system  performance  ratings. 
These  same  measures  are  hypothesized  to  also  correlate  highly 
with  the  ratings  of  system  work  and  the  effort  required  of  the 
controller  to  handle  the  system  load  and  meet  system  criteria. 
System  load  measures  of  the  number  of  aircraft  handled  and 
system  airspace  sector  size  and  geometry  are  hypothesized  to 
correlate  highly  with  system  load  factor  ratings. 


SYMBOLS 

P  System  performance 

Pdv  System  performance  measures  in  decivals,  dv. 

dv  A  decival,  equal  to  10  log1Q  0.,/X^J  ,  where  X  is  a  measured 

system  evaluation  factor  and  X-|/Xq  is  the  ratio  of  change  measured 
in  that  factor  during  the  period  when  it  is  being  observed  and  rated. 

PR  System  performance  rating. 

R  An  equal  interval  numerical  method  of  rating  of  system  factors 
(performance,  work,  load,  etc.)  ranging  from  ten  at  the  highest 
downward  to  one  at  the  lowest  on  an  equal  interval  scale.  The 
one-tenth  power  of  ten  scale  has  values  of  between  units 

and  10±0*01  between  tenths  of  rating  units  as  shown  in  figure  1. 

L  Load  on  the  system. 


Ldv  System  load  measure  in  decivals,  dv 


Lr  System  load  ratings 


W  System  work  Including  human  effort  required  to  handle  system 
load  and  meet  system  performance  criteria. 

Wdv  System  work  measure  including  human  effort  required  to  handle 

system  load  and  meet  system  performance  criteria  in  decivals,  av. 

WR  System  work  rating  including  the  human  effort  required  to  handle 
system  load  and  meet  system  performance  criteria. 

EQUATIONS:  SYSTEM  EVALUATION  HYPOTHESES 


Pdv  -  PR  ’  10 

(D 

Ldv  -  10  -  LR 

(2) 

wdv  -  10  -  WR 

(3) 

Pdv  *  10  log  £p/p) 

(4) 

Ldy  =  10  log  P^L/lJ 

(5) 

Wdv  -  10  log  gLw/3) 

(6) 

PR  -  10  +  10  log  0LP/P%*/1OoJ 

(7) 

LR  *  10  -  10  log  JfcL/L%*/103 

(8) 

WR  -  10  -  10  log  0W/W%*/1OJ) 

(9) 

♦As  these  are  magnitude  estimation  ratings  of  system  performance , 

P  load,  L,  and  work,  w  based  on  a  standard  optimum  rating  of  100%, 
as  shown  in  figure  1,  they  are  percentage  ratings.  Therefore, 
in  this  equation  percentage  ratings  must  be  divided  by  100.  pPhis 
provides  the  fractional  expression  of  the  rating  which  is  hypothe¬ 
sised  to  be  equal  to  the  ratio  of  the  measured  values  of  the 
observed  system  performance,  system  work,  and  system  load  factors 
compared  to  the  measured  standard,  ultimate  1001  performance,  work, 
and  load  factors^J 


DISCUSSION 


An  equal  interval ,  one-tenth  power  of  ten  ( 1 0 * 1 ) ,  with  ten 
the  highest  rating  is  introduced  to  unify  both  the  rating  and  the 
measurement  of  system  factors  of  interest  during  system  evaluation. 
The  quantitative  value  of  the  interval  between  units  of  this  scale 
is  designed  to  be  constant.  Throughout  the  scale ,  it  is  always 
plus  or  minus  the  one-tenth  power  of  ten  (10^*^)  of  the  system 
factor  being  rated  or  measured. 

Ten  times  the  logarithm  base  ten  is  termed  a  decival,  dv. 

The  scale  is  further  divided  into  tenths  so  that  one-tenth  of  the 
distance  between  units  is  always  equal  to  plus  or  minus  the 
one-hundredth  power  of  ten,  (10^0*01)  of  the  factor  being  rated  or 
measured . 

Ten  times  the  logarithm  of  these  one-tenth  unit  distances  or 
0.1  dv  is  termed  a  centival,  cv.  A  centival  is  hypothesized  to 
approximate  the  smallest  perceived  and  therefore  rateable  difference 
in  any  system  factor  being  rated. 

A  verbal  rating  is  assigned  to  each  unit  of  the  scale  to  provide 

a  qualitative,  general  descriptive  statement  of  the  type  and  degree 
of  rated  and  measured  system  performance,  work,  and  load  found 
during  the  system  evaluation. 

The  method  does  not  require  or  depend  on  the  use  of  10  times 
logarithms  of  system  measures  and  ratings.  However,  it  is  recom¬ 
mended  for  adoption  and  general  use  as  it  provides  a  simple  useful 


way  of  dealing  with  the  quantitative  statement  of  the  value  of 


the  underlying  basic  equal  interval  scale.  The  key  is  the  equal 
interval  written  and  communicated  as  powers  of  10,  i.e.,  10^0,1 


and  10±0*01, 


etc. 


Ten  times  the  logarithm  of  these  powers  of  10  is  only  a 
shorthand  way  of  referring  to  them  as  the  decival,  dv  and  the 
centival,  cv.  These  units  have  the  advantage  of  being  plus  or 
minus  so  many  decivals  or  centivals  which  are  easily  perceived  as 
being  equal  intervals.  These  intervals  represent  equal  fractions 
(  £k  X/X)  of  change  and  equal  percentages  of  change  when  multiplied 
by  100. 

Thus,  the  centival  or  decival  provides  a  simple  way  of  communicating 
equal  intervals  of  change,  increasing  or  decreasing  in  relation  to 
the  baseline  or  standard  condition. 


The  standard  or  baseline  system  conditions  of  qualitatively 
optimum  system  performance,  work,  and  load  required  to  meet  system 
criteria  are  quantified  at  100%,  zero  decivals  and  a  rating  of  10. 

In  the  Air  Traffic  Control  System,  these  system  criteria  or  ultimate 
goals  are  the  safe,  orderly  efficient  flow  of  air  traffic. 

The  basic  nature  of  the  system  evaluation  scale  is  to  depict 
changes  from  the  standard  system  condition  in  equal  intervals  of 
system  performance,  work,  and  load.  As  system  load  increases  100*1 
or  one  decival  (1.0  dv)  so  does  the  system  work  required  increase 
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100’1  or  1.0  dv;  however,  system  performance  has  degraded  10”0,1 
or  *>1.0  dv.  This  relationship  between  rated  and  measured  system 
load,  system  work,  and  system  performance  is  continuous  through¬ 
out  the  scale. 

For  example,  as  load  increases  to  lO***3  or  3  dv,  so  does 
system  work  or  effort  required  of  the  operator  representing  a 
doubling  of  each  from  the  baseline  measured  and  rated  conditions. 

In  turn,  system  performance  has  changed  by  10  0,3  or  minus  3  dv. 

This  represents  a  halving  of  system  performance  from  the  baseline 
systems  measured  and  rated  conditions. 

The  system  rating  R  has  also  changed  3  rating  points  to  a  7 
representing  a  good  system.  A  further  change  of  3  dv  increase  in 
system  load  and  work  represents  another  doubling  of  each  and  further 
halving  of  system  performance,  -3  dv.  The  system  rating  R  has 
decreased  a  further  3  rating  points  to  a  4  representing  an  unsatis¬ 
factory  or  bad  system. 

The  qualitative  verbal  rating  is,  of  course,  an  approximation 
of  the  quantitative  measured  values  which  will  be  determined  by 
system  tests.  It  may  prove  to  be  a  perfect  fit  of  language  to 
measured  results.  By  use  during  system  tests,  the  descriptive 
language  may  be  calibrated  and  changed  very  easily,  if  necessary, 
until  the  fit  is  proven  reliable. 

The  verbal  rating  may  be  the  least  important  part  of  the  system 
evaluation  process.  Even  so,  it  is  the  bottom  line  method  of 
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communicating  test  results  by  qualitatively  verbalizing  the 
results  of  the  evaluation  of  a  system. 

The  important  parts  of  the  system  evaluation  process  are  the 
simultaneous  measurement  and  rating  of  system  load,  work,  and 
performance.  The  potential  discovery  and  proof  of  the  hypothesized 
interrelationships  among  these  system  factors,  their  rating  and 
their  measurement  depends  on  the  use  of  an  equal  interval  rating 
and  measurement  scale. 

The  scale  must  possess  other  attributes  along  with  being  equal 
interval  throughout.  It  must  communicate  qualitatively  and 
quantitatively  the  standard  baseline  condition.  A  percentage 
approach  with  100%  assigned  to  the  standard  and  the  word  optimum 
along  with  a  rating  number  R  of  10  satisfies  these  requirements. 

It  has  the  further  advantage  of  providing  a  magnitude  estimation 
psychopyhsical  scaling  technique  basis  to  the  scale.  The  late 
Dr.  S.  S.  Stevens  originated  and  proved  the  power  of  magnitude 
estimation  (Reference  No.  1).  The  percentage  figures  communicate 
this  feature.  Additional  descriptive  language  is  throughout  the 
scale  to  reflect  commensurate  quantitative  changes  assigned.  All 
of  these  scale  features  are  useful  not  only  for  the  system  ratings, 
but  also  for  the  measurements  of  system  performance,  work,  and  load. 
The  percentage  and  decival  changes  in  system  measures  in  relation 
to  the  measured  and  rated  baseline  condition  can  also  be  established 
by  the  system  test.  The  relationship,  or  multiple  correlation,  of 
these  measures  to  the  verbal  and  numerical  ratings  may  then  be 
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established.  This  system  evaluation  process  provides  a  common 
baseline  and  a  common  scale.  Most  important  is  the  common 
relationship  of  measures  and  ratings  in  time  as  they  are  both 
obtained  simultaneously  throughout  the  system  test. 

At  this  point ,  a  properly  designed  and  conducted  system 
evaluation  as  discussed  in  the  PAA  Handbook  for  Experimenters 
(Reference  No.  2)  will  provide  results  anchored  to  a  qualitative 
definition  of  what  is  optimum,  excellent,  very  good,  good,  fair, 
poor,  or  bad  system  performance.  Along  with  this  will  be  established 
what  constitutes  optimum,  light,  moderate,  considerable,  extensive, 
great,  or  very  great  system  work  and  load. 

All  of  the  above,  at  this  time,  are  unknown  qualitative  and 
quantitative  relationships  among  system  performance,  system  work, 
and  system  load  factors.  They  can  be  established  by  employing  this 
method  during  dynamic  system  test  and  evaluation. 


SYSTEM  PERFORMANCE 


Figure  1.  An  example  of  an  equal  interval  rating  and  measurement  scale 
system  evaluation  of  performance,  work,  and  load. 
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